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Central Queensland University (CQU) encapsulates many of the recent changes to 
Australian universities. These changes include the imperative to diversify funding sources, 
the expansion of international education, the blurring of modes of study and the 
proliferation of online and other technologically based teaching and learning. This paper 
canvasses several of the issues framing current and potential strategies for evaluating the 
effectiveness of Central Queensland University's open and distance education provision. 
These issues include the institution's ongoing search for its identity; historically grounded 
practices and assumptions around open and distance education; changing demographics; 
expectations of contemporary university students and teachers; and evident tensions 
around the commercialization of some elements of the University's operations. The 
associated strategies are designed to respond to these issues at the same time as 
promoting diversity, equity and sustainability in the institution's open and distance 
education offerings. In combination, these issues and strategies derive from implicit - and 
too often unexamined - assumptions about which kinds of evaluation are viewed as 
'legitimate' and about who gets to make those judgments. The paper concludes by 
considering some of the key implications of current evaluation practices and the conceptual 
framework for understanding what is seen as 'legitimate' evaluation (and by whom) in 
contemporary Australian universities' open and distance education offerings, and the 
potential role of evidence-based practice in reinvigorating that debate. 

A BRIEF OVERVIEW OF CQU 

CQU is an Australian multi-campus regional university with an increasing focus on 
internationalizing its student population. Founded in 1967, it became a university in its 
own right in 1991 (Cryle,1992). Currently CQU has thirteen (13) campuses including five in 
regional Queensland, four in major Australian metropolitan centres and four campuses in 
other countries. In 2003 at CQU, 7 261 students (34%) were designated as 'distance 
education' or 'external students', while 1187 students (5.5%) were designated as 
"multimodal" or "internal and external" students (Luck, Jones, McConachie & Danaher, 
2004, p. 5). The remaining 12 903 students (60.5%) were deemed to be 'face-to-face' or 
'internal' students. Total student numbers more than tripled between 1990 and 2003, 
rising from 6000 to 21000 approximately. This increase was accompanied by an increasing 
diversity of the student profile. Up until the early 1990s, CQU's student population was 
approximately 50% internal and 50% external enrolments, and the majority of students 
were Australian. With the creation of the international campuses in Sydney, Melbourne, 
Brisbane and the Gold Coast during the 1990s, international students now form nearly 
40% of CQU's student population and they originate from 121 countries (UniNews Weekly, 
9 January 2004; cited in Luck et al., 2004, pp. 4-5). Moreover, the proportion of school 
leavers and mature age students has shifted dramatically, so that in 2003 only one-fifth of 
new students are recent school leavers (p. 5). 

EVALUATING TEACHING and LEARNING in a CHANGING ENVIRONMENT 



A major organizational response to evaluation processes occurred in 2001 when the 
University implemented a new policy for evaluation of the quality of teaching and learning. 
The document System-Wide Student Evaluation: Evaluation Guide (Central Queensland 
University, 2000), sets out guidelines for evaluation. These guidelines focus on a particular 
approach to evaluation, namely on surveys of students' opinions about their experience of 
teaching and course presentation. This evaluation approach was developed to show that 
the University has processes to deal with quality issues and has robust public 
accountability mechanisms. This evaluation information is also available for other 
purposes, for example: 

• to teaching staff for professional self assessment; 

• for staff appraisal and development; and 

• to support teachers' applications for promotion and tenure. 

The policy guidelines (p. 5) clearly state that a student satisfaction survey is only one 
evaluation tool, and that teaching staff should collect evidence by other means, for 
example: a journal to record reflection on practice and self-evaluation; students' journals 
outlining their reflection about the effectiveness of their own learning process; peer or 
external expert review of courses; head of school review of courses; and educational 
expert reviews. Staff are encouraged to compile the results of such personal, professional 
evaluation processes in a 'Teaching Portfolio'. 

However it is the 'official' student satisfaction survey process which is seen by many staff 
as the legitimate form of evaluation, because it has been given priority, is managed by the 
University and is done well. Evaluation conducted by individual teaching staff to improve 
learning is not seen to have the same status, and there is no evidence to suggest that it is 
done systematically by all teachers (Nouwens, Ross, Thomson, Harreveld & Danaher, 
2004). This personal, unofficial and informal evaluation which is directed at improving the 
teaching process is a critical aspect of a professional approach to teaching (Smith & Lovat, 
2003). However the status of the official compulsory University surveys of student 
satisfaction, together with recent government pressure for both summative evaluation of 
teaching and learning and use of performance indicators as measures of quality and 
accountability (Department of Education, Science and Training, 2004), encourages the 
development of an evaluation culture that gives legitimacy to these perceptions of 
'objective' measures and denies the importance of approaches to evaluation that have the 
potential to be more effective in improving teaching and learning. The critical reflection 
that follows begins with an analysis of the current 'high status' - and 'high stakes' - 
evaluation processes. Issues arising from a recent staff forum on evaluation are identified 
and used to inform this analysis. 

UNIVERSITY-WIDE TEACHING and LEARNING EVALUATION AT CQU 

Within CQU, Evaluation Services (ES) is a unit designated to provide course evaluation and 
teaching evaluation services and to maintain policy and standards for evaluation. ES 
facilitates the development and distribution of approved survey instruments, provides a 
collection service for surveys and provides data entry, analysis and reporting services. 
Course and teaching evaluations within CQU have been separated into four questionnaire 
models (see Table 1). The questions are suited to a variety of delivery modes including 
classroom and distance online evaluations. In addition to these formal. University-wide 
course and teaching evaluations that gather students' opinions of courses and teaching 
while they attend the University, the Evaluation Services unit also conducts the Graduate 
Destination Survey (GDS). The GDS is the annual questionnaire issued to graduates of all 
Australian universities to determine employment statistics for recent graduates. The GDS 
also includes questions (Course Experience Questionnaire survey) that summatively 
evaluate the quality of programs of study by seeking students' perceptions of the quality of 
their whole-of-program learning experience after they graduate. 



Table 1: Survey questionnaire models 



Student Evaluation of 
Courses (SEC) (Central 
Queensland University, 
2003a) 


Internal (face-to-face) 


Questionnaire: 20 closed 
questions (13 core and 7 
optionals chosen from a 
question bank) plus 3 open- 
ended questions 


Flexible (online or 
external education) 


Questionnaire: 20 core, closed 
questions and 3 open-ended 
questions 


Student Evaluation of 
Teaching (SET) (Central 
Queensland University, 
2003b) 


Internal (face-to-face) 


Questionnaire: 20 closed 
questions (10 core and 10 
optional questions) plus 3 
open-ended questions 


Flexible (online or 
external education) 


Questionnaire: 10 core, closed 
questions and 3 open-ended 
questions 



Issues in evaluating teaching and learning at CQU 

A forum with teaching staff about evaluation of teaching and learning at CQU (Nouwens et 
al., 2004) identified eight issues that are described briefly in the following paragraphs: 

• Response rates and representativeness of surveys; 

• Survey fatigue; 

• Student perceptions of effectiveness; 

• Cultural interpretations; 

• Student diversity; 

• Time-related pedagogical considerations; 

• Teaching appraisals; and 

• Student anonymity and trust. 

RESPONSE RATES AND REPRESENTATIVENESS OF SURVEYS 

Completion of teaching (SET) and course evaluation (SEC) questionnaires is not 
compulsory. Response rates of over 80% of course enrolment have been achieved from 
face-to-face classes. However, online surveys generally used with distance education 
students result in very low response rates, less than 20%. The low response rate for 
distance education students raises questions about bias in the responses. The responders 
may be a self-selecting group; for example, successful students or students who had 
difficulty with the course or the teacher. 

Survey fatigue 

Students may be requested to complete both course evaluations and teaching evaluations 
in four courses in the same week. Students' comments indicate they are over-surveyed 
and, because participation is not compulsory, there is a growing reluctance to complete 
them. Whether or not this affects students' judgements and the quality of their responses 
is open to argument and highlights the need for investigation into 'survey fatigue' (Luck et 
al., 2004). 

Student perceptions of effectiveness 

Forum participants also expressed concerns that some students doubt the effectiveness of 
the surveys for improving the quality of their courses and that this may influence the 
quality of their responses. Survey results are quarantined until after grades are certified so 
the SET/SEC information cannot be used to improve the learning experience of the 
surveyed cohort of students. There was concern that students may perceive evaluation as a 
University marketing tool, rather than an opportunity to identify strengths and weaknesses 
in order to eliminate or reduce the latter. 

Cultural interpretations 

Culturally influenced interpretations of the survey instruments and interpretations of the 



evaluation process itself may lead significant groups in the student population to respond 
differently to the same questions. Analysis of responses indicates that the wording and 
intention of the questions are interpreted differently through different cultural lenses. 

Student diversity 

Many CQU students study part-time, in distance mode and have work and family 
commitments. On the other hand, a significant number of students study full-time, on- 
campus, with easy access to learning resources. These different groups may be expected to 
respond differently to questions relating to issues such as study workload. Such 
demographic factors are not identified in the survey questions. 

Time-related pedagogical considerations: Research shows that the timing of questions can 
influence results (Kahnemann, 2000). Significant learning experiences and the average 
intensity and recency of these experiences are likely to bias responses to questions about 
the quality of learning experiences. Some approaches to teaching seek to challenge 
students to think for themselves and take responsibility for their own learning. Evidence 
from the staff forum suggests that the use of student-centred teaching approaches (such 
as Problem Based Learning) could result in poor scores on questions relating to items like 
course organization and support from students who prefer 'transmission' teaching. Surveys 
conducted before students are debriefed following completion of projects are more likely to 
produce such results. Perry (1988) points out that the development of complex attributes 
like commitment to ethical behavior and a preparedness to take responsibility for one's 
own learning occur in different rates in each student. Development is slow, complex and 
probabilistic (Knight & Yorke, 2003); it is not something that teachers can control, though 
appropriate pedagogy can scaffold and accelerate development. 

Teaching Appraisals 

Teaching staff at the forum expressed concern that Faculty supervisors. Deans and 
promotion committees may use the SET and SEC reports as authoritative indicators of the 
quality of staff performance. There was a perception that the scores on survey questions 
were used by these people as an objective measure of staff performance, and that 
qualitative information about performance was not seen to have the same level of 
legitimacy. 

Student anonymity and trust: Anonymity of student identity in SEC and SET data is 
designed to encourage students to voice concerns without fear of academic retribution. 
However, anonymity also makes it difficult to use the survey reports for formative 
improvement of teaching and courses because improvement of the teaching/learning 
process often requires a detailed understanding of the contexts that produced the data, 
and the identity of the learners is central to that understanding. This view is supported by 
Curzon-Hobson (2002), who suggests that critical, dialogical learning that is central to the 
development of professional graduate attributes requires a pedagogy of trust between 
teachers and learners. 

AN EVALUATION FRAMEWORK 

The evaluation policy and processes that have been developed at CQU are part of a pattern 
of ongoing responses to the complex changes occurring in teaching and learning at CQU. In 
this context, traditional approaches to the evaluation of teaching and the quality of courses 
have presented a range of challenging issues identified in the aforementioned staff forum 
(Nouwens et al., 2004). New approaches are needed to develop a robust and effective 
evaluation framework that reflects complex dilemmas inherent in the disembodiment of 
knowledge via formalised teaching and learning (Snowden, 2002). Here we are arguing 
that, in the field of higher education, evaluation should be considered in terms of three 
different contexts: (1) learners; (2) teaching and learning processes; and (3) 
institutionalised sites in which that teaching and learning take place. This three-fold 
evaluation framework is developed using Habermas' (1996) analysis of human cognitive 
interest that identifies such interests as emancipatory, practical and technical. 



Some understanding of the problems of evaluating teaching processes in complex 



organisations like universities may be gained from ideas emerging in the field of 
knowledge management. Snowden claims that "we always know.. .more than we have the 
physical time or the conceptual ability to say. I can speak in five minutes what would 
otherwise take me.. .a couple of hours to write down" (2002, p. 102). This notion that we 
lose something in the telling is crucial to an understanding of the evaluation process, 
because it invites reflection upon what is lost when we attempt to "disembody" (Snowden, 
2002) the teaching and learning experience by describing it in numbers and text in survey 
questionnaires (SEC, SET), interview transcripts or other evaluation tools. 

Yet large organisations and governments cannot operate without some degree of 
disembodiment of knowledge. To make timely and effective decisions, senior managers 
seek information that has been filtered from the rich and idiosyncratic detail of individuals' 
experiences. Thus, while teachers and learners develop understandings of the learning 
process that are based on specific, experiential knowledge and mutual relationships, 
aspects of which are difficult to generalise and articulate; managers develop a disembodied 
"organizational view derived from an abstract understanding of the task" (Warne, Ali, 
Linger & Pascoe, 2003, p. 307). 

Disembodiment of knowledge incurs two types of costs. First, to articulate or to document 
the experience of learning in an authentic way requires an investment of time and thus 
requires resources. For a student to describe a learning experience to support the process 
of personal self-reflection may require only a few rough notes on paper. The same student 
would need to keep a more comprehensive journal to share the same learning experience 
in any detail with peers or with the class teacher. The greater the distance between a 
student's experience of learning and the recipient of that student's description of that 
experience, the more disembodied is the knowledge expressed, and the greater will be the 
cost of recording a faithful description of the experience. The second type of cost incurred 
in the disembodiment process is a reduction of authenticity in the description. As Snowden 
(2002) notes above, we cannot say or record everything we know, because some 
knowledge remains tacit and no description can accurately replicate a learning experience. 

The information environments of learners, teachers and managers are very different, and 
so are their interests and purposes. Learners in higher education often need support to 
develop skills for reflecting upon, evaluating and improving personal learning experiences 
(Knight & Yorke, 2003). For teachers, evaluation is at the heart of curriculum processes 
(Smith & Lovat, 2003). Teachers seek information to evaluate learners' progress and the 
contribution of the teaching-learning relationships to that learning. Governments and 
university managers seek to ensure that the institution operates effectively and efficiently 
with transparent accountability for the disbursement of publicly funded resources 
(Department of Education, Science and Technology, 2004). All three perspectives are 
legitimate. Effective evaluation to monitor and improve learning requires an appropriate 
balance among the interests of the learners, the teachers and management. 

We contend that improvement of the quality of higher education depends on a commitment 
to evidence-based practice that fully involves all participants in the education process. This 
requires the intentional development of an institutional culture of evaluation that 
recognizes and integrates the learning, teaching and management interests in evaluation 
and gives appropriate emphasis to each interest. Habermas (1996) provides a useful 
framework for analyzing these three interests in evaluation and integrating them in a 
culture of evaluation. In human social activity, he identifies three areas of cognitive 
interest, namely: 

• technical cognitive interest relates to the use of knowledge in exercising predictive 
control over objectivised processes, over natural processes and work, to produce the 
goods that sustain life and society. For our purposes in this paper, this is the prime 
interest of the manager of the educational process. 

• practical cognitive interest relates to the use of knowledge in making meaning, and 
for the preservation and expansion of mutual understanding required for practical 
consensual action. It is built upon cultural tradition, particularly on the use of 
language to create and share meaning and to encourage organised human activity. 
Teachers and learners operate together in this area of interest. 




• emancipatory cognitive interest relates to the self-theories (Yorke & Knight, 2004) 
and use of knowledge in sharing power over human activity. It involves the capacity 
of individuals to reflect independently and responsibly on social activity and to act 
autonomously (Smith & Lovat, 2003). This represents the learners' interests in the 
educational process. 

Each of these three areas of cognitive interest adopts a particular mode of inquiry to 
produce legitimate knowledge, and each of these modes of inquiry follows particular logic 
and methods. These modes of inquiry determine the legitimacy of evaluation in each area 
of cognitive interest and for each interest group (Habermas, 1996). 

Table 2: Relationship of cognitive interest to inquiry modes in evaluation 



Cognitive interest 


technical 


practical 


emancipatory 


Interest group 


management 


Teachers-learners 


learners 


Inquiry mode 


empirical-analytical 


historical-hermeneutic 


social-critical 



Table 2 can be expanded to explore the nature of evaluation for each interest group, to 
indicate the appropriate intent of evaluation in each case and to indicate what evaluation 
approaches could legitimately be used. This expansion is based on Habermas' work (1996) 
and on Snowden's study (2002) of how organisations seek to manage productively the 
internal flow and development of knowledge that produces improvement. The evaluation 
framework in Table 3 suggests that the three interests in evaluation are both legitimate 
and very different in purpose and method. Thus the kinds of evaluation a manager requires 
to support decision-making should not be used as substitute measures for evaluation of 
teaching/learning processes, or to determine whether effective learning has occurred. 

Table 3: A framework of purposes and approaches to evaluation 



technical interest in 
evaluation 


practical interest in evaluation 


emancipatory interest in 
evaluation 


develop predictable, 
controlled learning 
systems 


promote collaborative and 
productive teacher-learner 
relationships 


achieve transformative and 
responsible personal 
development 


provide explicit, 
transparent policy, 
procedures, training 


develop a community of learning 
with clear, shared objectives 


develop trust in learning 
networks, shared values and 
experiences 


obtain generalisable, 
reliable information to 
make decisions about 
others 


interpret and share meaning to 
guide and improve learning 


reflect critically on personal 
activities to construct 
effective self-theories 


use mainly summative 
evaluation: best 
practice, measurement, 
ranking, performance 
indicators 


use mainly formative evaluation: 
good practice, dialogue, 
collaboration, feedback, 
improvement 


use mainly reflective self- 
evaluation of personal 
action: self-reporting, 
journals, portfolios 


legitimacy of evaluation 
based on objective, 
analytical processes and 
institutional authority 


legitimacy of evaluation based 
on authority negotiated and 
shared between teachers and 
learners, and on discipline and 
learning community standards 
of good practice 


legitimacy of evaluation 
based on personal values, 
beliefs and commitment and 
the authority of each 
learner's personal 
experience 




































DISCUSSION 



The complexities of technology-based education in a multi-mode, multi-campus university 
present a challenge for the evaluation of the quality of teaching and learning. The issues 
identified at the CQU staff forum, issues that were described earlier in this paper, indicate a 
need for more robust and holistic evaluation frameworks than the simple concepts of 
evaluation that have been used in traditional higher education. Such evaluation 
frameworks should recognize the interests of managers, teachers and learners in the 
educational process. They should understand the corresponding evaluative interests of 
these groups, namely technical, practical and emancipatory, and recognize that each area 
legitimately involves the application of different evaluative processes. 

The SEC/SET surveys described earlier are 'technical' and largely summative in that they 
can usefully indicate to managers that students have a problem with some aspect of a 
course (e.g., workload, clarity of objectives). However the reports give little detailed 
information about the nature of the problem or how to fix it. From the point of view of CQU 
student respondents, the surveys are summative because they can have no impact on 
learning in their current course; accordingly the staff forum indicated that students might 
see these surveys as providing information for marketing courses rather than for improving 
them. From the University's point of view they are summative accountability measures 
required by the government. The university is seen to support this formal, technical type of 
evaluation with staff and resources (see Figure 1), and has integrated requirements for this 
aspect of evaluation into procedures to support policy (Central Queensland University, 
2003a, 2003b). 

While policy guidelines to evaluation (Central Queensland University, 2000) invite teaching 
staff to develop evaluation portfolios to support the SEC/SET surveys with authentic 
descriptions of the teaching/learning process, there is little support available to implement 
this. 

The policy guidelines do not mention the role of learner self-evaluation in the evaluation 
process, although a capacity for lifelong learning is nominated as an attribute that the 
University seeks to develop in its graduates. Thus the emphasis at CQU is on the use of 
evaluation for management purposes, and little explicit organizational support is provided 
for evaluation to improve teaching or learning. Indeed staff forum comments indicate that 
at least some teachers have a perception that evaluation for management purposes has 
become a de facto standard for teaching evaluation. 

If universities are to meet demands to develop effective professional capabilities in 
graduates, then student-focused emancipatory interest in evaluation should receive more 
extensive and explicit attention in teaching (Scott, Yates & Wilson, 2001). Such capabilities 
require the explicit development of emotional intelligence and robust understanding of self 
that grows from a capacity to reflect critically on one's experience, motivation, 
responsibility, commitment and sense of identity (Holmes, 2000; Knight & Yorke, 2003; 
Perry, 1988). However, traditional approaches to higher education have made content the 
official, explicit discourse, while personal development was a hidden agenda that was not 
formally recognized in program or course management. Thus the official processes used to 
evaluate learning focus attention on the content of what is learned, not on the personal 
development of capabilities that higher education now seeks to make an explicit outcome 
of learning. For this reason, the evaluation of learning needs to be extended from the 
traditional focus on content to include the evaluation of the effectiveness of learners' 
personal development, and that requires the use of evaluation processes that align with 
emancipatory interests. 

The practical interest in evaluation relates to the relationships between teachers and 
learners. Like any professional behaviour, effective teaching and curriculum development 
depend on evaluation (Smith & Lovat, 2003). Such evaluation ranges from momentary 
interpretation of individual and class responses with consequential feedback, to more 
reflective processes that use notes on teaching events, documented reflection in journals 
and scholarship and research in teaching. Effective higher education invites learners to 



engage in a journey of inquiry into the world and self. Curzon-Hobson (2000) argues that 
this spirit of inquiry grows in a trusting relationship and a sense that the teacher will both 
care about and challenge the learner: "students must sense the teacher's willingness and 
passion to hear" (p. 269). This capacity to hear each learner and respond to that learner's 
concerns is central to this practical interest in evaluation, to the teacher obtaining an 
understanding of the learner's understanding and responding appropriately. Developing a 
productive teacher/learner relationship involves more than determining whether a student 
has "confidence in the teacher that the content of a program is 'up to date' and that 
methods of assessment are 'fair' or 'valid' (Curzon-Hobson, 2000, p. 268). Yet these are 
matters that management accountability surveys seem intent on determining. 

Figure 1: Holistic evaluation of teaching and learning 

Teacherself- 




In summary, a holistic evaluation framework must give appropriate weight to 
management, teacher and learner interests in education. Figure 1 indicates that the focus 
of attention in the practice of evaluation at CQU has been on management interests. The 
increasing complexity of university operations requires that the organization give more 
explicit attention to teacher evaluation of the teaching-learning process (using methods 
like peer- and self-evaluation), and the development in learners of self-evaluation skills 
and the development in the university of a learning environment conducive to the practice 
of such skills. 

CONCLUSION 

Using this evaluation framework to analyse the concerns expressed about the course and 
teaching evaluation processes at CQU suggests that a clear distinction be made among 
evaluation for management purposes on the one hand, and evaluation of teaching/learning 
processes and student self-evaluation of learning on the other. The SEC/SET surveys are 
effective educational management tools for identifying strengths and areas of weakness in 
course planning and delivery that require further investigation. However, while they are 
effective evaluations for management purposes, they are neither authentic evaluations of 
the teaching-learning process nor effective evaluations of the individual learning that has 
occurred. The development of a culture of evaluation of teaching and learning is an issue 
for teaching professionals, it can be supported and promoted by institutional 
encouragement and use of reflective teaching portfolios and evaluative or research focused 
scholarship of teaching. Studies should also investigate the effectiveness of the learners' 
emancipatory interests and explore the impact of the education process on development of 
self-reflection and self-evaluation capabilities that provide a foundation for high-level 
attributes that graduates are expected to develop. 

Such a holistic framework for evaluation is required to establish the legitimate role of 
different approaches to evaluation, and to justify a more balanced allocation of time and 
resources to improving students' learning. A robust conceptual framework is essential in 
the complex and changing environment that is CQU. The complexity of teaching has 
increased with multi-mode teaching, the use of online, multimedia and videoconference 
technologies, the inclusion of international students and increasing numbers of mature 



students on more campuses,. The University must ensure that evaluation processes related 
to its core activity — learning — keep pace with this change. 
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